{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 maxmium()与minmium()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " maximum()用于限制最小值,也即是说，将一个tensor中小于指定值的元素替换为指定值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=3, shape=(10,), dtype=int32, numpy=array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = tf.range(10)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=5, shape=(10,), dtype=int32, numpy=array([4, 4, 4, 4, 4, 5, 6, 7, 8, 9], dtype=int32)>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.maximum(a, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=9, shape=(3, 4), dtype=int32, numpy=\n",
       "array([[8, 2, 4, 1],\n",
       "       [9, 5, 4, 7],\n",
       "       [6, 5, 8, 6]], dtype=int32)>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = tf.random.uniform([3,4], minval=1, maxval=10, dtype=tf.int32)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=11, shape=(3, 4), dtype=int32, numpy=\n",
       "array([[8, 4, 4, 4],\n",
       "       [9, 5, 4, 7],\n",
       "       [6, 5, 8, 6]], dtype=int32)>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.maximum(b, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "minium()方法与maximum()方法想法，用于限制一个tensor的最大值，即将tensor中大于指定值的元素替换为指定值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=13, shape=(10,), dtype=int32, numpy=array([0, 1, 2, 3, 4, 5, 6, 6, 6, 6], dtype=int32)>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.minimum(a, 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=15, shape=(3, 4), dtype=int32, numpy=\n",
       "array([[6, 2, 4, 1],\n",
       "       [6, 5, 4, 6],\n",
       "       [6, 5, 6, 6]], dtype=int32)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.minimum(b, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果要同时限制一个tensor的最大值和最小值，可以这么做："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=19, shape=(3, 4), dtype=int32, numpy=\n",
       "array([[6, 4, 4, 4],\n",
       "       [6, 5, 4, 6],\n",
       "       [6, 5, 6, 6]], dtype=int32)>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.minimum(tf.maximum(b,4),6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这种同时调用minmium()和maxmium()的方法不够便捷，所以TensorFlow中提供了clip_by_value()方法来实现这一功能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 clip_by_value()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "clip_by_value()底层也是通过调用minmium()和maxmium()方法来实现同时限制最大值、最小值功能，我们现在来感受一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=9, shape=(3, 4), dtype=int32, numpy=\n",
       "array([[8, 2, 4, 1],\n",
       "       [9, 5, 4, 7],\n",
       "       [6, 5, 8, 6]], dtype=int32)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=23, shape=(3, 4), dtype=int32, numpy=\n",
       "array([[6, 4, 4, 4],\n",
       "       [6, 5, 4, 6],\n",
       "       [6, 5, 6, 6]], dtype=int32)>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.clip_by_value(b,4,6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 relu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "relu()方法将tensor最小值限制为0，相当于tf.maxmium(a,0),注意，relu()方法在tf.nn模块中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=27, shape=(10,), dtype=int32, numpy=array([-5, -4, -3, -2, -1,  0,  1,  2,  3,  4], dtype=int32)>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = tf.range(-5,5,1)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=28, shape=(10,), dtype=int32, numpy=array([0, 0, 0, 0, 0, 0, 1, 2, 3, 4], dtype=int32)>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.relu(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=32, shape=(3, 4), dtype=int32, numpy=\n",
       "array([[-8, -1, -4,  7],\n",
       "       [-6, -3,  2, -8],\n",
       "       [ 5,  6,  2,  5]], dtype=int32)>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = tf.random.uniform([3,4],minval=-10, maxval=10, dtype=tf.int32)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=33, shape=(3, 4), dtype=int32, numpy=\n",
       "array([[0, 0, 0, 7],\n",
       "       [0, 0, 2, 0],\n",
       "       [5, 6, 2, 5]], dtype=int32)>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.nn.relu(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 cli_by_norm()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "cli_by_norm()方法是根据tensor的L2范数（模）和给定裁切值按比例对tensor进行限幅。这种方法可以在不改变方向的前提下，按比例对向量进行限幅。\n",
    "我们先手动实现这一过程，先定义一个向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=39, shape=(2, 3), dtype=float32, numpy=\n",
       "array([[ 9.93618 , 10.367402,  9.617832],\n",
       "       [ 8.890949,  9.650288,  9.430309]], dtype=float32)>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = tf.random.normal([2,3],mean=10)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后求这个向量的L2范数，也就是向量的模："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=44, shape=(), dtype=float32, numpy=23.66054>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = tf.norm(a) \n",
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "向量处理模，就可以将向量缩放到0到1范围："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=45, shape=(2, 3), dtype=float32, numpy=\n",
       "array([[0.41994733, 0.43817267, 0.4064925 ],\n",
       "       [0.3757712 , 0.4078642 , 0.39856696]], dtype=float32)>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a1 = a / n\n",
    "a1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对向量限幅时，例如限制在10范围内："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=47, shape=(2, 3), dtype=float32, numpy=\n",
       "array([[4.1994734, 4.3817267, 4.064925 ],\n",
       "       [3.757712 , 4.078642 , 3.9856696]], dtype=float32)>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a2 = a1 * 10\n",
    "a2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "clip_by_norm()方法实现的就是上述步骤："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=63, shape=(2, 3), dtype=float32, numpy=\n",
       "array([[4.1994734, 4.3817267, 4.064925 ],\n",
       "       [3.757712 , 4.0786424, 3.9856696]], dtype=float32)>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.clip_by_norm(a,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然，cli_by_norm()方法内部还做了一个判断：如果给定的裁切值大于tensor的模，那就不会去对tensor进行修改，依旧返回tensor本身。继续上面例子，a的模为25.625225，如果给定的裁切值大于这个值，就不会对a进行限幅："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=79, shape=(2, 3), dtype=float32, numpy=\n",
       "array([[ 9.936181, 10.367402,  9.617832],\n",
       "       [ 8.890949,  9.650288,  9.430309]], dtype=float32)>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.clip_by_norm(a,26)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 clip_by_global_norm()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在梯度更新等诸多场景中，需要同时综合多个参数（tensor）进行梯度更新，这时候，clip_by_norm()就满足不了需求了，所以就有了cip_by_global_norm()方法。cip_by_global_norm()方法限幅原理与clip_by_norm()是一样的，都是综合范数和给定的裁切值进行限幅，不同的是，cip_by_global_norm()方法方法计算范数时是综合给定的多个tensor进行计算。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注：clip_by_global_norm()方法用于修正梯度值，控制梯度爆炸的问题。梯度爆炸和梯度弥散的原因一样，都是因为链式法则求导的关系，导致梯度的指数级衰减。为了避免梯度爆炸，需要对梯度进行修剪。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下面三个向量为例，同时进行限幅："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=85, shape=(3,), dtype=float32, numpy=array([8.257121, 7.466969, 8.756236], dtype=float32)>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t1 = tf.random.normal([3],mean=10)\n",
    "t1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=91, shape=(3,), dtype=float32, numpy=array([10.112761, 10.555879,  9.646121], dtype=float32)>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2 = tf.random.normal([3],mean=10)\n",
    "t2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=97, shape=(3,), dtype=float32, numpy=array([9.884818, 8.648524, 9.125227], dtype=float32)>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t3 = tf.random.normal([3],mean=10)\n",
    "t3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_list = [t1,t2,t3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先计算全局L2范数,计算公式为：\n",
    "global_norm = sqrt(sum([L2norm(t)**2 for t in t_list]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "global_norm = tf.norm([tf.norm(t) for t in t_list])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假设给定裁切值为25："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: id=121, shape=(3,), dtype=float32, numpy=array([7.4725804, 6.757504 , 7.9242725], dtype=float32)>,\n",
       " <tf.Tensor: id=124, shape=(3,), dtype=float32, numpy=array([9.151909, 9.552924, 8.729607], dtype=float32)>,\n",
       " <tf.Tensor: id=127, shape=(3,), dtype=float32, numpy=array([8.945623, 7.826795, 8.258204], dtype=float32)>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[t*25/global_norm for t in t_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<tf.Tensor: id=148, shape=(3,), dtype=float32, numpy=array([7.47258  , 6.7575035, 7.9242725], dtype=float32)>,\n",
       "  <tf.Tensor: id=149, shape=(3,), dtype=float32, numpy=array([9.151908, 9.552924, 8.729606], dtype=float32)>,\n",
       "  <tf.Tensor: id=150, shape=(3,), dtype=float32, numpy=array([8.945623 , 7.8267946, 8.2582035], dtype=float32)>],\n",
       " <tf.Tensor: id=136, shape=(), dtype=float32, numpy=27.624733>)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.clip_by_global_norm(t_list,25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算结果是一样的，不过clip_by_global_norm()返回两个值，分别是各向量限幅后的返回值列表、全局范数。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "study_python",
   "language": "python",
   "name": "study_python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
